[Content Addressable] Bundler Local Cache and Lockfile by OughtPuts · Pull Request #178 · Shopify/rubygems

OughtPuts · 2026-08-18T17:50:24Z

TL;DR

This PR updates the bundler install flow for locally cached gems to support content addressable gem naming.

Description

This PR builds on the Bundler remote-install support for content-addressed gems.

It writes the gem’s content address and platform to Gemfile.lock, then parses those values back into LazySpecification during a subsequent install. This lets Bundler identify the content-addressed .gem in vendor/cache and install it with bundle install --local, without contacting the remote source again.

The same content-addressed name is used for the installed gem directory in both remote and local installation paths, ensuring lockfile round-trips produce a consistent on-disk layout.

Tests

Updates made to lockfile_parser_spec, content_addressable_spec and ext_spec to cover the changes.

Tophatting

Start a local rubygems.org server, that contains a content-addressable gem.
Set your gemfile to point to the local server and a content-addressable gem that is available via that server.
Run bundle install from the directory that contains your gemfile, using a Ruby version compatible with the content addressable gem.
Confirm that the Gemfile.lock contains a content address:

GEM
  remote: http://localhost:3000/
  specs:
    ca_test_harriet (0.3.5-d26b82f6) arm64-darwin-23

PLATFORMS
  arm64-darwin-23

DEPENDENCIES
  ca_test_harriet (= 0.3.5)

CHECKSUMS
  ca_test_harriet (0.3.5-d26b82f6)

BUNDLED WITH
  4.1.0.dev

Remove any locally installed gem files other than the cached files (normally in the /cache/ directory)
Re-run bundle install --local. Confirm the content addressable gem installs correctly still and that it uses the correct content addressable name.

I've performed this tophat locally and it's all looking good! ✅

Copilot

Pull request overview

This PR adds Bundler/RubyGems support for content-addressable gem artifacts end-to-end, including compact-index v2 test infrastructure, lockfile encoding/parsing, and ensuring local cache resolution works with content-addressed filenames.

Changes:

Extend specification/name tuple objects to carry content_address, and update resolution & lockfile serialization/parsing accordingly.
Add Artifice compact-index v2 helpers to model content-addressable gems in specs, including platform metadata injection.
Add/adjust specs to cover lockfile formatting/parsing and local-cache round-trips for content-addressable gems.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
spec/support/builders.rb	Build helper now supports producing content-addressable gem files (Ruby ABI–scoped).
spec/support/artifice/helpers/compact_index.rb	Checksum calculation updated to use `spec.full_name` gem filenames.
spec/support/artifice/helpers/compact_index_v2.rb	New compact-index v2 Artifice helper with content-addressable gem support and metadata injection.
spec/support/artifice/helpers/compact_index_cooldown.rb	Removed (superseded by v2 helper).
spec/support/artifice/compact_index_v2.rb	New Artifice activation entrypoint for the v2 helper.
spec/support/artifice/compact_index_cooldown.rb	Removed (tests now use compact_index_v2).
spec/other/ext_spec.rb	Add coverage for `Gem::NameTuple#lock_name` and `Bundler::LazySpecification#to_lock` with content addresses.
spec/install/gemfile/content_addressable_spec.rb	New integration specs for content-addressable install, caching, and lockfile round-trips.
spec/install/cooldown_spec.rb	Switch cooldown tests to use the compact-index v2 Artifice setup.
spec/bundler/remote_specification_spec.rb	Rename “platform” to “suffix” in test setup and add content-address expectation coverage.
spec/bundler/override_spec.rb	Ensure overrides stubs include `content_address`.
spec/bundler/lockfile_parser_spec.rb	Add parsing coverage for lockfile lines with a trailing content address token.
spec/bundler/endpoint_specification_spec.rb	Add coverage for CA suffix handling via platform metadata and remote fetching behavior.
lib/rubygems/specification.rb	Include `content_address` in `Specification#name_tuple`.
lib/rubygems/safe_marshal.rb	Permit `Gem::NameTuple` to safely marshal/unmarshal `@content_address`.
lib/rubygems/name_tuple.rb	Add `content_address` field on `Gem::NameTuple`.
lib/bundler/stub_specification.rb	Propagate `content_address` when building StubSpecifications.
lib/bundler/source/rubygems.rb	Preserve content address when swapping remote specs after download.
lib/bundler/rubygems_integration.rb	Populate spec `content_address` from `Gem::Package` when available.
lib/bundler/rubygems_gem_installer.rb	Call RubyGems’ content-address assignment hook when supported.
lib/bundler/rubygems_ext.rb	Compatibility shims for older RubyGems plus lock name behavior for content-addressed tuples.
lib/bundler/resolver.rb	De-dupe resolved specs including `content_address` to avoid collapsing distinct CA variants.
lib/bundler/remote_specification.rb	Add `content_address` and include it in `full_name` when applicable.
lib/bundler/match_platform.rb	Prefer compatible content-addressable candidates when available.
lib/bundler/lockfile_parser.rb	Parse trailing content-address token on spec lines and propagate into LazySpecification.
lib/bundler/lazy_specification.rb	Track `content_address`, include it in `full_name`, and emit it in lock output.
lib/bundler/fetcher.rb	Treat the 3rd token as a “suffix” and pass through to endpoint/remote specification constructors.
lib/bundler/endpoint_specification.rb	Add CA suffix parsing via metadata-provided platform and fetch remote specs using CA suffix when present.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

OughtPuts · 2026-08-19T12:34:40Z

-      spec = @specs[full_name]
+      lock_name = Gem::NameTuple.new(name, version, platform).lock_name
+      @specs_by_lock_name ||= @specs.values.to_h {|candidate| [candidate.lock_name, candidate] }
+      spec = @specs[full_name] || @specs_by_lock_name[lock_name]


This allows us to correctly find the CA gem spec by either the full_name OR the lock_name after the recent naming changes, when storing the checksum.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

jenshenny

Looking nice! I just had a question on the way we're storing the CA and the platform in the lockfile.

I'm curious on why the platform is being stored for the lock name and have CA outside instead of the CA as the lock name and the platform outside. The CA would provide uniqueness and be clear that it's the suffix of the .gem file but I might be missing something.

OughtPuts · 2026-08-20T15:49:10Z

Looking nice! I just had a question on the way we're storing the CA and the platform in the lockfile.

I'm curious on why the platform is being stored for the lock name and have CA outside instead of the CA as the lock name and the platform outside. The CA would provide uniqueness and be clear that it's the suffix of the .gem file but I might be missing something.

Yeah I see what you mean. It's just because the regex in lockfile_parser already expects to parse the platform in that position, so it made most sense to me to leave platform where it was and add the CA afterwards as an additional placing.

I'm happy to be outvoted though if you feel strongly.

jenshenny · 2026-08-20T16:15:23Z

Yeah I see what you mean. It's just because the regex in lockfile_parser already expects to parse the platform in that position, so it made most sense to me to leave platform where it was and add the CA afterwards as an additional placing.

Ah I see! I still feel that switching the CA and the platform make most sense to me because it's consistent throughout the system that the content address is the version token for skinny gems and the platform is the version token for fat ones. Having that discrepancy in the lockfile is a bit confusing to me...

It's a bit more work to handle both in $4 of the regex, but we're doing the same in remote specification and storing it as a suffix or version token instead.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

jenshenny

Approving to unblock! Looks good, just some other minor comments on other backwards compat cases

jenshenny · 2026-08-22T13:53:36Z


    def name_tuple
-      Gem::NameTuple.new(@name, @version, @platform)
+      Gem::NameTuple.new(@name, @version, @platform, content_address: @content_address)


I think this is causing jruby to fail, I think we could add something in rubygems_ext to omit the additional arg based on RG version.

Nice thanks - I've added the shim based on whether initialize accepts content_address or not rather than tieing to a version, thinking this will make it more robust, just in case.

Append platform to content-addressable gems entries in lockfile Support content addressable gems in lockfile and local cache

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

OughtPuts marked this pull request as draft August 18, 2026 17:50

OughtPuts requested a lite review from Copilot August 18, 2026 17:50

Copilot started reviewing on behalf of OughtPuts August 18, 2026 17:51 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Comment thread lib/bundler/lockfile_parser.rb

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch 3 times, most recently from 474c200 to e884790 Compare August 19, 2026 09:25

OughtPuts requested a lite review from Copilot August 19, 2026 09:26

Copilot started reviewing on behalf of OughtPuts August 19, 2026 09:26 View session

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from e884790 to 9f26cc0 Compare August 19, 2026 09:30

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Comment thread lib/bundler/lockfile_parser.rb Outdated

Comment thread lib/bundler/stub_specification.rb

Comment thread lib/bundler/rubygems_integration.rb

Comment thread lib/bundler/rubygems_gem_installer.rb Outdated

Comment thread lib/bundler/lazy_specification.rb Outdated

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 9f26cc0 to b985f74 Compare August 19, 2026 09:46

OughtPuts requested a lite review from Copilot August 19, 2026 09:47

Copilot started reviewing on behalf of OughtPuts August 19, 2026 09:50 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Comment thread spec/install/gemfile/content_addressable_spec.rb Outdated

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from b985f74 to 7179457 Compare August 19, 2026 09:58

OughtPuts requested a lite review from Copilot August 19, 2026 09:58

Copilot started reviewing on behalf of OughtPuts August 19, 2026 09:59 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

Comment thread lib/bundler/lockfile_parser.rb Outdated

Comment thread lib/bundler/lockfile_parser.rb Outdated

OughtPuts marked this pull request as ready for review August 19, 2026 12:29

OughtPuts commented Aug 19, 2026

View reviewed changes

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 7179457 to 582815e Compare August 19, 2026 12:49

OughtPuts requested a lite review from Copilot August 19, 2026 12:49

Copilot started reviewing on behalf of OughtPuts August 19, 2026 12:49 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

OughtPuts requested review from girachawda and jenshenny August 19, 2026 14:17

jenshenny mentioned this pull request Aug 19, 2026

Add content addressable gems support ruby/rubygems#9773

Draft

5 tasks

OughtPuts force-pushed the feature-branch-ca-changes-rubygems branch from aad98a3 to 0caed62 Compare August 19, 2026 15:56

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 582815e to 0f79ede Compare August 20, 2026 13:44

jenshenny reviewed Aug 20, 2026

View reviewed changes

Comment thread lib/bundler/lockfile_parser.rb Outdated

Comment thread lib/bundler/lockfile_parser.rb Outdated

jenshenny reviewed Aug 20, 2026

View reviewed changes

Comment thread lib/bundler/rubygems_integration.rb Outdated

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 66e5cca to 97380de Compare August 21, 2026 16:08

jenshenny requested a balanced review from Copilot August 22, 2026 12:37

Copilot started reviewing on behalf of jenshenny August 22, 2026 12:38 View session

Copilot AI reviewed Aug 22, 2026

View reviewed changes

jenshenny approved these changes Aug 22, 2026

View reviewed changes

Support content addressable gems in lockfile and local cache

ce7e319

Append platform to content-addressable gems entries in lockfile Support content addressable gems in lockfile and local cache

OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 97380de to ce7e319 Compare August 24, 2026 12:18

OughtPuts requested a balanced review from Copilot August 24, 2026 13:20

Copilot started reviewing on behalf of OughtPuts August 24, 2026 13:20 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

OughtPuts merged commit a7d2ef8 into feature-branch-ca-changes-rubygems Aug 24, 2026
87 of 110 checks passed

Uh oh!

Conversation

OughtPuts commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Description

Tests

Tophatting

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

OughtPuts Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jenshenny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OughtPuts commented Aug 20, 2026

Uh oh!

jenshenny commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jenshenny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jenshenny Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

OughtPuts Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

OughtPuts commented Aug 18, 2026 •

edited

Loading

OughtPuts Aug 24, 2026 •

edited

Loading